I was able to slightly improve indentation of comments with > normal mode command (for comment started from 1st column only 1 space is used instead of shiftwidth).
But the bug you've found in insert mode still persists. I'll try to experiment with cinkeys.

Thanks. I look forward to it. I think this is a big step in the right direction, but worry if I found an arguably minor regression, what other possibly larger regressions have I missed? Hopefully whatever could resolve this issue would mitigate at least some risk of regression elsewhere.

It seems that cinkeys doesn't help with the issue found or at least I'm unable to find proper values for it.
Also I found that with current fix = normal mode command changes indentation of comments back & forth at root scope at each run. For example:

# comment
# another comment
  # mistakenly indented comment

after gg=G looks like:

  # comment
  # another comment
# mistakenly indented comment

after .:

# comment
# another comment
  # mistakenly indented comment

and so on.

I'll experiment with indentexpr. Looks like it's possible to combine it with C-indent by using cindent function. So, solution may be perform handcrafted indentation for edge cases & call cindent function otherwise. This also could make possible to skip indentation in here-strings.